Dual discriminator adversarial distillation for data-free model compression

Knowledge distillation has been widely used to produce portable and efficient neural networks which can be well applied on edge devices for computer vision tasks. However, almost all top-performing knowledge methods need access the original training data, usually a huge size is often unavailable. To tackle this problem, we propose novel data-free approach in paper, named Dual Discriminator Adversarial Distillation (DDAD) distill network without of any data or meta-data. specific, use generator create samples through dual discriminator adversarial distillation, mimics data. The not only uses pre-trained teacher’s intrinsic statistics existing batch normalization layers but also obtains maximum discrepancy from student model. Then generated are train compact under supervision teacher. proposed method an closely approximates its teacher network, using Extensive experiments conducted demonstrate effectiveness CIFAR, Caltech101 ImageNet datasets classification Moreover, extend our semantic segmentation tasks several public such as CamVid, NYUv2, Cityscapes VOC 2012. best knowledge, first work generative model based large-scale ImageNet, Experiments show that outperforms baselines distillation.